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ABSTRACT 

This'' paper describes the development of standardized 
evaluation procedures for Pasco County's insecvice teacher education. 
Since 1983, legislation requires teacher education centers in Florida 
to report attainment data based on measurable objectives* Using a 
classification system adapted from the "Rand Change Agent Study," 
directors and supervisors categorized each unit of their inservice 
components as one of seven types of inservice. An appropriate 
evaluation procedure designed to provide evidence of the increased 
knowledge and skill of participants was identified for each inservice 
type. These activities culminated in the generation of an inservice 
classification system with a concomitant evaluation strategy with two 
types of instruments— achievement tests and rating seal's. The 
"Inservice Evaluation Handbook" was created to provide assistance in 
developing these instruments. Although implementation difficulties 
emerged much progress was 'made in inservice "evaluation. 
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Abstract 



Attenpts to bring a sense of uniformity of measurement to Inservice 
evaluation began with the classification of all inservice components. 
Using a classification system adapted from*the Rand Change' Agent Stucfr , 
directors and supervisors were asked to categorize each unit of*their 
inservice components in accordance with one of seven types of inservice. 
An appropriate evaluation procedure designed to provide evidence of the 
'increased knowledge and skill of participants was then identified for 
each type of inservice. These activities culminated in the generation 
of an inservice classification system with a concomitant evaluation 
strategy. 

* 

Basically, the product evaluation procedures for each type of inservice 
component called for two types of instruments -yfachievement tests and 
rating scales. In other words, the focus or coftent of an instrument 
(what it measured) would vary contingent upon component classification, 
butTthe nature of the -4m rumen t (how it measured) was fixed. A hand- 
book was developed to provide assistance in the development of these 
evaluation instruments. 



BACKGROUND 

Pasco County's current inservice evaluation practices 
are the result of both state and local influences. In 1983, 

'legislative changes significantly impacted evaluation by 
requiring that teacher education centers report attainment 
data based on measurable objectives, the emphasis was 
clearly on outcome in terms of participant learning. At the 
same time r withi-n the county there was a growing concern for 
the manner in which inser vices were evaluated. In some 

i instances, participant gain was assessed quite rigorously, 
while in others, it was not. Therefore, in reexamining its 
evaluation of inservice, Pasco County sought to establish 
uniform (standard) procedures that would provide evidence of 
participant gain. a 

METHODOLOGY 

f 

Several important steps were taken toward this end of 
"standardized evaluation procedures". Components were 
grouped (classified), appropriate evaluation procedures iden- 
tified, and a handbook developed to facilitate implementa- 
tion of the new evaluation procedures. Each of these steps 
is described in detail below. 

CLASSIFICATION OF COMPONENTS " 

Every inservice component was examined to determine** 
commonalities for groupi-ng purposes. A classification 
system adapted from the Ralid Change Agent .Study was used 
to facilitate this process. Directors and supervisors were 
asked to categorize their inservice components in accordance 
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with the following definitions: 



I The unit is designed to motivate participants by 
providing them with ideas and/or information 
about a subject or trend. It* is inspirational in 
tone. 

II The unit is designed to get a group of partici- 
pants together to share ideas (brainstorm) ^bout 
a topic in order to assimilate the ideas received 
from a unit I type activity. 

III The unit is designed to compare the information 
assimilated from units I and II to existing con- 
ditions in order to determine needs. 

IV Once units I, II, and III types have been done, 
the purpose of unit IV is to plan. 

V The unit is designed to provide personnel 
training in order to implement a plan. The 
performance skills are identified, and strategies 
to teach the skills ar^ designed. 



VI The unit is designed to review and evaluate 

activities described in units I - V. The purpose 
is to modify the steps described in units I - V. 

VII There is a purpose for the unit other than what 
? is described in I - VI. 

The results of this classification exercise v ^re 
reviewed and tabulated. Type seven 7 (VII), the "other" 
category, was dropped as a viable category. 
IDENTIFYING EVALUATION PROCEDURES , 

The next step entailed forming a committee to identify 
appropriate evaluation procedures for each type of inservice 
being offered in the county. Three district office staff 
members (the Director of Research and Evaluation Services, 
the Supervisor of Staff Development, and the Specialist for 
Program Evaluation Services) comprised the committee. The 
following schema which they proposed (and the county 
adopted) is presented in Table I. 



TABLE 1: IHSERVICE TYPE AND DESIGNATED EVALUATION PROCEDURES 



IHSERVICE 4 EVALUATION • EVALUATION 

TYPE PURPOSE FOCUS PROCEDURE 



II 



III 



IV 



VI 



To inform the participants 
about a subject or trend 
affecting organisational 
development • 

To have participants develop 
and share ideas (brains torn) 
pertaining to a subject or 
trend affecting organiza- 
tional development • 

To have participants analyze 
information for the identi- 
fication of the specific 
requirements (needs) which 
must be met in order to 
adequately plan for organ- 
isational development. 

To have participants develop 
objectives, strategies, and 
evaluation procedures (a plan) 
designed to effect change in I 
organisational development. 

To provide participants with 
the knowledge or skill re- 
quirements necessary to im- 
plement strategies designed 
to effect change v in organi- 
sational development. 

To have participants analyse 
evaluation results for the 
^identification of needed 
modification to objectives, 
strategies, or evaluation 
procedures . 



Knowledge gained by the 
partici pants pertaining 
to the subject or trend , 



Contribution to the 
group process by each 
participant* . 



Contribution to the 
group process by each 
participant . 



Contribution to the 
group process by each 
participant. 



***** 



Knowledge gained by the 
participants pertaining to 
requirements to implaaent 
strategies designed to 
effect change in organisa- 
tional development. 

Contribution to the 
group process by each 
participant. 1 - 



A pre/posttest designed 
using a valid and reliable 
objective test. 



A rating of the quality of, 

the contributing* f° r each 
participant. >5fte component 
coordinator must establish 
the performance criteria. 

A rating of the quality of 
the contributions for each 
participant. The component 
coordinator must establish 
the performance criteria. 



A rating of the quality of 
the contributions for each 
participant. The component 
coordinator must establish 
the performance criteria. 

A pre/posttest design using 
a valid and reliable 
objective test. 



A rating of the "quality of 
the contributions for each 
participant. The component 
coordinator must establish 
the performance criteria* 



DEVELOPMENT OF THE HANDBOOK . • 

t 

One final step was undertaken to ensure implementation 
of the county-adopted evaluation procedures. A handbook was 

4 

created to provide assistance with the development of evalua- 
tion instruments for inservice components. The goal was to 

00m 

develop a "cookbook" (to be used by educators of varying 
backgrounds and abilities) which would result in instruments 
of high quality. 

Since clearly stated and measurable objectives are fund- 
amental to product evaluation, the first section of the hand- 
book was devoted to writing behavioral objectives. The ap- 
proach was to move from objectives defining the specific 
competencies to be gained, to a test custom-fitted to those - 
objectives. 

As discussed earlier^ appropriate evaluation procedures for 
each type of inservice component were identified. Basically, 
the designs called for two types of instruments - achievement 
tests and rating scales. In other words, if a component was , 
identified as type I or V (information/testing), the evaluation 
instrument would be an achievement test. Similarly, if a compo- 

0 

nent was identified as type II,. Ill, IV-, or VI {performance- 
based), the "evaluation instrument would , be a rating scale. The 
focus or content of the instrument ( what it measured) would 
vary contingent upon component classification, but the nature of 
the instrument ( how it measured) was fixed. 

Therefore, subsequent sections of the handbook discussed 
the mechanics of developing achievement tests and rating scales. 
The issues of planning the test, writing and refining items, 
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test form/physical appearance, and reliability and validity 

* 9 
also addressed. 

„ The final section of the handbook described procedures 
for submitting instruments for approval. This review pro- 
cess was a quality control measure for ensuring that all 
insetvice evaluation instruments met certain minimum stan- 
dards. Prior to staging an inservice, the proposed ev*lua- 
tion instrument was to be submitted (for approval to the 
District Office, Department of Research and Evaluation 
Services along with the following; ^ ' 

1. a list of specific learner objectives; 

2. an outline of the inservice's content? 

3. a description of the delivery format (lecture, dis- 
cussion, "hands-on", etc; 

4. an identification of the target group (recipients 
of the inserv.ice); 

5. the delivery date (when the inservice would be 
held) . 

IMPLEMENTATION DIFFICULTIES 

* 

During the months following county adoption of the new 
evaluation procedures, several implementation difficulties 
emergedt Basically, three areas proved troublesome - 
fcommitment to serious evaluation, proficiency at writing 
behavioral objectives, and early submission requirements. 
Each area is discussed separately below. 
COMMITMENT TO SERIOUS EVALUATION 

In the initiai stages, the usual resistance to change 
was encountered. Many directors and supervisors complained 



about the inordinate amount of time that would be required 
to develop suitable instruments. Another frequent, complaint 

* • 

focused on the problems surrounding use of consultants to 
deliver inservice; Either the consultants did not always 
deliver the intended content, or they had their own manner 
of assessing the inservice - not necessarily. consistent with 
the county procedures. 

DIFFICULTY WRITING OBJECTIVES 

t 

Once those responsible for developing evaluation instru- 
ments got down to the task at hand , other problems emerged. 
For many, it was difficult moving from broad program objec- 
tives to specific performance objectives. Related. to this 
was a weakness in the ability to .identify key elements of a 
learner behavior (for measurement purposes). 
EARLY SUBMISSION REQUIREMENTS 

The' greatest difficulty encountered regarding inservice 
evaluation instruments stemmed from early submission require 
ments. In our county, all components were due for approval 
in March. Notification of approved components did not occur 

i 

\ * 

until August. Some components with vague or very broadly 
stated objectives were approved. Later, as the actual 
staging date approached, the content was developed. 
Sometimes^ it was developed by a person other than the one 
who wrote the objectives. This often led to situirtions 
where the content did not fit the" objective^. On other 
occasions, difficulties arose in attempting to write test 
items for components wit!j poorly stated objectives. 
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CONCLOSIONS/RECi 



)ATIOHS 



Despite the noted difficulties, much progress was made 
last year in ttte area of inservice evaluation. Perhaps the,* 
greatest factor contributing to successful implementation of 
the new evaluation procedures was the staging of wo* h ses- 
sions. The purpose of those sessions was to afford an oppor- 
tunity to practice the mechanics* of writing test items. 
Directors and supervisors attended the work sessions armed,, 
with the broad objectives and any material relevant to' an 
inservice component of their choice. During tine sessions 
they were able to work as a group* On certain concepts, but 
the specific content was different for each participant. 

The establishment of a review process for approving • 
instruments (or recommending revisions) was critical to the 1 
maintenance of an effective evaluation system. It also made 
a strong statement regarding administrations commitment to 
sound evaluation practices. t 
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